Mounted display goggles for use with mobile computing devices

ABSTRACT

A head mounted display system for use with a mobile computing device includes a main body configured to be worn on a human head. A lens assembly within the main body includes a first lens configured to focus vision of a wearer on a first area of a display of the mobile computing device when the mobile computing device is secured to the main body. A first contact point disposed in a fixed position relative to the first lens is configured to contact a surface of the display when the mobile computing device is secured to the main body. The first contact point is detectable by the mobile computing device and usable by the mobile computing device to derive a position of the display relative to the at least one lens.

REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/221,848 filed Jul. 28, 2016, now U.S. Pat. No., entitled“Soft Head Mounted Display Goggles for Use with Mobile ComputingDevices”, which is a continuation of U.S. patent application Ser. No.15/148,891 filed May 6, 2016, now U.S. Pat. No. 9,599,824, entitled“Soft Head Mounted Display Goggles for Use with Mobile ComputingDevices”, which is a continuation of U.S. patent application Ser. No.14/625,602 filed Feb. 18, 2015, now U.S. Pat. No. 9,377,626, entitled“Remote Control Augmented Motion Data Capture”, which claimed priorityunder 35 U.S.C. §119 to U.S. Provisional Application Ser. Nos.62/060,996, filed Oct. 7, 2014, and 61/941,294, filed Feb. 18, 2014,both entitled “Mobile Virtual and Augmented Reality System and Use,” thecontents of which are expressly incorporated herein by reference.

The present application is also related to U.S. patent application Ser.No. 14/687,104 filed Apr. 15, 2015, now U.S. Pat. No. 9,176,325 issuedNov. 3, 2015 entitled “Soft Head Mounted Display Goggles for Use withMobile Computing Devices” which is a continuation of U.S. patentapplication Ser. No. 14/625,591 filed Feb. 18, 2015, now U.S. Pat. No.9,274,340 issued Mar. 1, 2016 entitled “Soft Head Mounted DisplayGoggles for Use with Mobile Computing Devices”.

The present application is also related to U.S. patent application Ser.No. 14/625,603 filed Feb. 18, 2015, entitled “Interpupillary DistanceCapture Using Capacitive Touch”, and U.S. patent application Ser. No.14/687,121 filed Apr. 15, 2015, entitled “Interpupillary DistanceCapture Using Capacitive Touch.”

NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. This patent document may showand/or describe matter which is or may become trade dress of the owner.The copyright and trade dress owner has no objection to the facsimilereproduction by anyone of the patent disclosure as it appears in thePatent and Trademark Office patent files or records, but otherwisereserves all copyright and trade dress rights whatsoever.

BACKGROUND

Field

This disclosure relates generally to wearable computers, and morespecifically to goggles which receive a mobile computing device such asa smartphone to provide a mobile virtual and augmented reality system,whereby a user can experience and control virtual reality (VR),augmented reality (AR), and stereoscopic experiences, such as threedimensional (3D) and 360° movies and computer games.

Any discussion of the prior art throughout this specification should inno way be considered as an admission that such prior art is publiclyknown or forms part of common general knowledge in the field.

In the 1960s, Ivan Sutherland presented a virtual 3D world to usersusing an early vector cathode ray tube (CRT) head mounted display.Tracking was performed by a set of either mechanical or ultrasonicsensors. A general purpose computer processed the tracking data, while aspecial purpose graphics processor made the appropriate perspectivetransforms on scene data. Sutherland wrote, “No availablegeneral-purpose computer would be fast enough to become intimatelyinvolved in the perspective computations required for dynamicperspective display.”

Since that time, the graphics hardware industry has grown and matured.With the rise of the video game industry, there is now a commoditizedmarketplace for high performance graphics chipsets. Such chipsets enablealmost any general-purpose computer to run 3D game engines and allowthese machines to “intimately” participate in real-time perspectivedisplay. These chipsets are now in mobile computing devices, such ascurrent smartphones, bringing 3D game engines to these smaller devices.

Head mounted displays (HMDs) have provided gateways into variousaugmented and virtual realities, and have been used in many industriesin addition to gaming as a means of allowing hands free and immersiveviewing of computer generated and filmed (e.g., 360° cameras) content.However, these displays were typically manufactured in low volumes, werebuilt for a customer base of researchers and niche applicationdevelopers, and cost thousands, if not tens of thousands, of dollars.There have been some steps towards commodity virtual reality displaysfor gaming, such as the Nintendo Virtual Boy™, but these products havebeen commercially unsuccessful. A variety of relatively low cost mobileHMDs (MHMDs) have been available in the $1000 and lower price point,beginning with models such as the Sony Glasstron™, Virtual I/OiGlasses™, and continuing with some models today.

There is a need for a more ergonomic and user-friendly system for MHMDsthat leverage the sophistication and capabilities of current mobilecomputing devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are described below with referenceto the accompanying diagrammatic drawings, in which:

FIG. 1 is a rear perspective view of head mounted display goggles inaccordance with one embodiment of the invention with a mobile computingdevice poised to be received therein;

FIG. 2 is an exploded perspective view of components of the gogglesshown in FIG. 1;

FIG. 3a is a perspective view of the goggles shown in FIG. 1 fitted on aperson;

FIG. 3b is a side view of the goggles shown in FIG. 1 fitted on aperson;

FIG. 4a is a perspective view of one embodiment of the goggles shown inFIG. 1 illustrating exemplary functional design features;

FIG. 4b is a perspective view of one embodiment of the goggles shown inFIG. 1 illustrating use of an external frame to secure and position themobile computing device;

FIG. 4c is a perspective view of one embodiment of the goggles shown inFIG. 1 illustrating use of an internal frame to secure and position themobile computing device;

FIG. 5 is a top down view of one embodiment of the goggles shown in FIG.1 fitted on a person and illustrating stereoscopic viewing achievedthrough the lenses;

FIG. 6 is a perspective view of an exemplary lens assembly for thegoggles shown in FIGS. 1 and 2;

FIG. 7a and FIG. 7b are perspective views of an exemplary remotecontroller for use with the goggles shown in FIG. 1;

FIG. 8a and FIG. 8b are perspective views of an alternative remotecontroller for use with the goggles shown in FIG. 1;

FIG. 8c is a plan view of a control face of a still further alternativeremote controller for use with the goggles shown in FIG. 1;

FIG. 9a is a perspective view of one embodiment of a remote controllerillustrating use of a remote controller accessory attachment port toattach a fiducial marker accessory;

FIG. 9b shows the same view with a lighted ball in place of the fiducialmarker;

FIG. 9c is a first person view of one embodiment of a remote controllerillustrating the use of the fiducial markers on a remote controlleraccessory to attach a virtual object;

FIG. 9d is a flow diagram describing one embodiment of a markerdetection process;

FIG. 10a is a perspective view of an exemplary embodiment of the lensassembly for the goggles shown in FIGS. 1 and 2;

FIG. 10b is an exploded view of the lens assembly of FIG. 10a showingmechanical components of the assembly;

FIG. 10c is a perspective view of the lens assembly of FIG. 10a showinga mechanical slide and lock system as well as a pair of stylusesextending therefrom;

FIG. 10d illustrates exemplary use of the styluses along with conductivematerial to create corresponding contact points on the mobile devicescreen;

FIG. 10e illustrates the relationship of the contact points CP with thelens fulcrums;

FIG. 10f is a flowchart diagram of a method for determining the positionof contact points and computing changes in software based on thepositions;

FIG. 11a is a perspective view of an MHMD illustrating one embodiment ofa fiducial pattern embedded into the visual appearance;

FIG. 11b illustrates using computer vision to detect the MHMD anddisplay virtual information, in this case an avatar;

FIG. 11c illustrates placement of virtual objects based on detection ofthe MHMD;

FIG. 11d is a flow diagram describing detection of a marker andplacement of a virtual object;

FIG. 12 illustrates functional application of foam material for a mainbody of the MHMD;

FIG. 13a illustrates an embodiment of the MHMD that includes additionalelectronic components as well as a side slot for inserting a mobilecomputing device;

FIG. 13b illustrates exemplary electrical components of an alternateembodiment of the MHMD with the main body in phantom;

FIG. 13c is a flow diagram of an exemplary sensor interface process forthe MHMD;

FIG. 14a illustrates computer vision detection of a known object usingthe MHMD;

FIG. 14b is a first person view illustrating virtual objects beingplaced in relation to a known object;

FIG. 14c is a flow diagram of a method for detection of a physicalobject and placement of a virtual object;

FIG. 15a is a perspective view illustrating use of point clouds todetermine user perspective and scale of physical environments;

FIG. 15b is a perspective view illustrating a virtual environment placedonto a physical environment based on point cloud data;

FIG. 15c is a flow diagram of a method for using point cloud data todisplay a virtual environment;

FIG. 16a is a perspective view illustrating interaction between mobilecomputing devices and a signal processing server;

FIG. 16b is a top view illustrating interaction between mobile computingdevices and a signal processing server;

FIG. 16c is a flow diagram of a method for interaction between mobilecomputing devices and a signal processing server;

FIGS. 17A and 17B are perspective views of a further embodiment of themobile head mounted display (MHMD) goggles of the present application;

FIGS. 18A and 18B show a person wearing the MHMD goggles of FIGS. 17Aand 17B in two different modes of operation;

FIG. 19 is a perspective exploded view of the MHMD goggles of FIGS. 17Aand 17B;

FIGS. 20A-20L are various orthogonal and sectional views of a soft mainbody of the MHMD goggles of FIGS. 17A and 17B, namely:

FIGS. 20A and 20B are front and rear perspective views, respectively,

FIGS. 20C and 20D are front and rear elevational views, respectively,

FIG. 20E is a top plan view,

FIG. 20F is a sectional view looking forward through a mobile computingdevice retention slot and taken along angled lines 20F-20F in FIG. 20E,

FIG. 20G is an alternative sectional view looking forward through amobile computing device retention slot and taken along angled lines20F-20F in FIG. 20E, while

FIG. 20H shows a smartphone centered within the retention slot bycompressible bumpers,

FIG. 20I is a bottom plan view,

FIG. 20J is a right side elevation view (the left side being identicalin this embodiment),

FIG. 20K is a vertical sectional view taken along line 20K-20K in FIG.20E, and

FIG. 20L is a vertical sectional view taken along line 20K-20K in FIG.20E showing an upper retention ridge of the retention slot;

FIG. 21A is a side elevation view of the MHMD goggles of FIGS. 17A and17B, and FIG. 21B is a horizontal sectional view through the gogglestaken along line 21B-21B of FIG. 21A;

FIGS. 22A and 22B are front and rear perspective views of exemplary lensassemblies for use in the MHMD goggles of FIGS. 17A and 17B;

FIG. 23 is a top elevation view of the main body of the MHMD goggles ofFIGS. 17A and 17B shown in phantom illustrating movement of the lensassemblies therein relative to a mobile computing device;

FIGS. 24A-24E are perspective and top plan views of an alternative lensassembly with a movable stylus for use in the MHMD goggles of FIGS. 17Aand 17B;

FIGS. 25A-25E are perspective and top plan views of a furtheralternative lens assembly with a movable stylus for use in the MHMDgoggles of FIGS. 17A and 17B;

FIGS. 26A and 26B are front and rear perspective views, respectively, ofan exemplary remote control for use with the MHMD goggles of FIGS. 17Aand 17B;

FIGS. 27A and 27B are top and bottom perspective views, respectively, ofan exemplary circuit board for using the remote control of FIGS. 26A and26B;

FIGS. 28A and 28B schematically illustrate a fully inflatableconfiguration of the MHMD goggles of the present application;

FIGS. 29A and 29B show a partially inflatable embodiment of the MHMDgoggles; and

FIG. 30A is a side view of an alternative MHMD body having a capacitivetouch slider on one side, and FIG. 30B is a sectional view much likeFIG. 20H showing the position of the slider relative to a smartphonewithin the goggles.

DETAILED DESCRIPTION

The present application provides an ergonomic and user-friendly headmounted display for producing virtual reality (VR), augmented reality(AR), and stereoscopic experiences, such as three dimensional (3D) and360° movies and games. The head mounted display includes soft gogglesthat conform to a wearer's face and include a slot for receiving andretaining a mobile computing device, such as a smartphone. A pair oflenses adjustably mounted within the goggles provide a stereoscopicimage of the display of the smartphone within the goggles. One or tworemote controls may be mounted to the goggles for additionalfunctionality.

The term “head mounted display” or HMD refers to any apparatus that canbe mounted on the head to provide the wearer a personal viewingexperience. Illustrated embodiments include goggles that are strappedaround the back of the head and have a main body which receives a mobilecomputing device therein. Although a HMD can be relatively cumbersome,each of the HMDs described herein are relatively lightweight andportable, and thus are referred to as mobile head mounted displays, orMHMDs.

The term “mobile computing device” refers to a portable unit with aninternal processor/memory and a display screen, such as a smartphone.Mobile computing devices can be smartphones, cellular telephones, tabletcomputers, netbooks, notebooks, personal data assistants (PDAs),multimedia Internet enabled cellular telephones, and similar personalelectronic devices that include a programmable processor/memory anddisplay screen. Such mobile computing devices are typically configuredto communicate with a mobile bandwidth provider or wirelesscommunication network and have a web browser. Many mobile computingdevices also include a rear-facing camera which provides additionalfunctionality when coupled with the MHMDs of the present application.

In the exemplary head mounted display HMD shown in FIGS. 1 and 2, a mainbody 10 may be fitted with a lens assembly 20, a strap 40 which securelyattached the main body to the user's head, a re-attachable remotecontroller 30, and an external mobile computing device 50 to be securedin the main body 10. The main body 10 as disclosed herein is easilyadapted to fit any of a number of mobile computing device 50 shapes andsizes, such as, but not limited to, the iPhone5™, the iPod touch™, theSamsung Galaxy4™, the Nokia 920™, or any other handheld visual mediaplayers.

As noted, a strap 40 may be used to securely attach the main body to theuser's head, as illustrated in FIG. 3a and FIG. 3b ; however, other oradditional means and methods may be used, such as various items andtechniques that are readily available for other goggles- andglasses-type products which may be applied to the main body 10. Forexample, the main body 10 could be incorporated into a helmet-likedevice which is secured to the top of the head without a strap.

The exemplary mobile computing device 50 as seen in FIGS. 4a and 5includes a central processing unit (CPU) (not shown), a screen 52, aback facing camera 54, and wireless communication functionality (notshown), and may be capable of running applications for use with thesystem. In some embodiments, an audio port 56, such as shown in FIG. 4a, may be included, whereby audio signals may be communicated with thesystem. The mobile computing device 50 may incorporate one or moregyroscopes, gravitometers, magnetometers and similar sensors that may berelied upon, at least in part, in determining the orientation andmovement of the overall MHMD. In some embodiments, the mobile computingdevice 50 may be a third party component that is required for use of thesystem, but is not provided by or with the system. This keeps cost downfor the system by leveraging the user's current technology (e.g., theuser's mobile computing device).

FIG. 4a illustrates a perspective view of one embodiment of theexemplary goggles shown in FIG. 1 illustrating exemplary functionaldesign features. As may be seen, a main body 10 is shown that has acompartment 18, which is sized to fit and secure a mobile computingdevice 50. The main body 10 is hollowed out to allow the securely fittedmobile computing device 50 screen 52 to be visible from the back side of(i.e., from behind) the main body 10, as seen in section in FIG. 5. Whena user puts the main body 10 over his or her head using the strap 40,the display screen 52 is visible within the hollow interior of the body.In the embodiment of FIG. 4a , the main body 10 has holes 16 and 12 thatallow access to the device's various ports and components while thedevice is secured within the main body 10. In the particular embodimentshown, hole 12 allows the mobile computing device's 50 camera 54 to befully utilized, and hole 16 allows access to the mobile computingdevice's 50 audio port 56 to allow the attachment of external audioperipherals such as headphones 80, although it should be noted that inother embodiments, other numbers, sizes, and positions of holes may beimplemented as desired. For example, small vent holes may be provided tohelp prevent fogging of lenses and the display screen 52 within the mainbody 10.

As also indicated, in the exemplary embodiment shown, the main body 10has a Velcro™ element 11 to allow the re-attachment of the remotecontroller 30 as shown in FIG. 1. Of course, in other embodiments, theuse of Velcro™ to re-attach the remote controller can be replaced (oraugmented) with any of various alternative attachment methods or means,such as clip 39 shown in FIG. 8b , positioned in a similar place or indifferent location(s) on the main body 10 or strap 40 as shown inFIG. 1. In the exemplary embodiment of FIG. 4a , the main body 10 hasreinforced slots 14 to allow the attachment of the strap 40 to the mainbody 10 as shown in FIG. 1; however, the method of attachment of thestrap 30 can be accomplished by any of various other methods ofattachment, such as, but not limited to, sewn-in, glue, snaps, hooks,tabs, or Velcro® magnetics, among others.

FIG. 4b is a perspective view of one embodiment of the exemplaryapparatus shown in FIG. 1 illustrating exemplary use of an externalmobile computing device frame 19 to secure and position the mobilecomputing device. The mobile computing device 50 can be fitted into themobile computing device frame 19 so as to allow the main body 10 toreceive mobile computing devices of different sizes. In other words, useof the common frame 19 may allow any of various sized mobile computingdevices to be used as desired, and the shape of receptacle within themain body 10 reliably receives the common frame 19.

FIG. 4c is a perspective view of one embodiment of the exemplaryapparatus shown in FIG. 1 illustrating use of an internal frame 51 tosecure and position the mobile computing device. “Internal” is meansthat the frame 51 is designed to reside within the main body 10. Themobile computing device may be inserted into the internal frame 51. Theinternal frame 51 may be rigid and a known shape which aids in centeringand leveling within the foam body. Alternatively, the internal frame 51is somewhat less compressible than the rest of the main body 10 so as tobetter center and level the mobile computing device, but is somewhatflexible so as not to detract from the otherwise soft and flexible mainbody. The “internal” frame 51 is shown in FIG. 4c both outside the body10 to illustrate its configuration and inside the body in its normalplacement. The use of an internal frame 51 may allow for incorporation,e.g., attachment or insertion, of mobile computing devices of differentsizes while properly positioning the device within the main body of theMHMD. In some embodiments, the use of spring tension parts (e.g., leafsprings) of the internal frame 51 a, 51 b, and 51 c may securely fit themobile computing device (not shown) into the main body.

Additional or alternative mechanisms as the frames 50, 51 are envisionedthat allow for similar functionality, such as, for example, the use ofan internal frame that operates as a toaster-like mechanism to allow themobile computing device to be inserted into the main body and click intoplace, wherein another push allows the device to be released.Furthermore, one or more internal frames may be provided, such as one todefine a pocket to retain the mobile computing device and another todefine channels within which are mounted the lens assembly 20.

FIG. 6 is a perspective view of an exemplary lens assembly 20 shown inFIGS. 1 and 2, according to one embodiment. As indicated, lens assembly20 contains two lenses shown in FIG. 6 as elements 21 a and 21 b. Insome embodiments, the lenses may be fixed in lens housings, exemplaryembodiments of which are illustrated in FIGS. 1 and 2. As also indicatedin FIG. 6, the lens housing are desirably attached to a lens assemblybase 23.

The lens assembly is located between the user 70 and mobile computingdevice screen 52, as illustrated in FIG. 5, which is a horizontalsectional view of one embodiment of the exemplary apparatus shown inFIG. 1 fitted on a person and illustrating stereoscopic viewing achievedvia the lenses. As may be seen, the main body 10 is worn with the user'seyes aligned with the lenses 21 a and 21 b so that the user 70 can lookthrough the lens to view the mobile computing device screen 52. Eachlens, e.g., of lenses 21 a and 21 b, may focus the user's vision S1 orS2 on a discrete (or respective) area of the mobile computing devicescreen L or R (left or right). Properly centering the user's visionthrough the lenses is particularly important in virtual realityapplications where simulation of natural vision in anartificially-generated world requires images of a known distance apartto be simultaneously presented to a user's eyes in order to properlyappear as “real” images.

The image on mobile computing device screen L is the left portion of thestereoscopic image, while mobile computing device screen R is the rightportion of the stereoscopic image. Video content which is stereoscopicmay be downloaded to the mobile computing device 50 to allow a person toperceive the images through the lenses 21 a, 21 b as one singlethree-dimensional image. Alternatively, stereoscopic display software orapps may be downloaded to the mobile computing device 50 and used toconvert any single image into one which is stereoscopic. Stereoscopicviewing allows creation of virtual reality (VR), augmented reality (AR),360 video, as well as 3D video.

FIGS. 7a and 7b are perspective views of exemplary remote controllerssuch as shown in FIG. 1, and FIGS. 8a and 8b are perspective views ofalternative remote controllers. In some embodiments, the remotecontroller 30 as illustrated in FIG. 7a , FIG. 7b , FIG. 8a , and FIG.8b , receives input from the user 70 (not shown) and communicates theinput to the mobile computing device 50 (not shown). While in someembodiments, wired means may be used to communicate between the remotecontroller and the mobile computing device, wireless communication ispreferred. For example, in some embodiments, a near-field wirelesscommunication protocol, such as Bluetooth, may be employed as a means tocommunicate to the mobile computing device 50; however WIFI is alsoconsidered as an alternative means of communication. More generally, invarious embodiments, any wireless (or wired) communication means orprotocols may be used as desired.

In the case of a wireless connection, the application running on themobile computing device 50 may use a method of detecting one or morecontrollers and determining if the application can or should connect tothe controllers based on the distance from the mobile device 50 usingthe signal strength of the remote controller. Alternatively, physicalinteraction with between the mobile computing device (or HMD) and acontroller (e.g. pressing or holding down a button) may signal that theyshould attempt to communicate with one another. In addition theapplication running on the device may connect to multiple controllersand provide distinct functionality to each controller connected. Inaddition the application running on the mobile device 50 may provide ameans of storing a record of controllers connected so that the systemcan ignore other controllers if needed, e.g., may be configured to storesuch a record in the memory of the mobile device 50.

FIG. 8a and FIG. 8b illustrate an embodiment in which the remotecontroller 30 comprises one or more buttons 32 a-32 f and/or one or moredirectional pads 34. In some embodiments, when the user 70 presses onone or more of the buttons 32 a-32 f or directional pad 34, the remotecontroller, e.g., a circuit board (not shown) included therein, may senda signal to the mobile computing device 50 corresponding to the button,direction, and/or possibly pressure. FIG. 8c illustrates an embodimentthat incorporates the use of distinct shapes for the buttons 32 a - 32 gas a means of allowing the user to feel for the button and determine thespecific button by shape without looking. The remote controller may alsoinclude a dedicated button that provides a specific function regardlessof the application being run, such as, for example, displaying theuser's camera feed on the mobile device.

In addition, in some embodiments, the remote controller 30 may beequipped with one or more motion sensing elements, e.g., one or moresensors for detecting movement, acceleration, orientation, and so forth,referred to herein generally as “motion detection.” Thus, for example,in some embodiments, the remote controller may include one or moremotion detection chip(s), e.g., 9-axis motion detection chips, althoughother numbers of motion-related axes may be used as desired. The remotecontroller 30 may communicate its current motion state (which mayinclude orientation) to the mobile computing device 50 according to somespecified criteria, e.g., at a specified frequency, e.g., one or moretimes per second, or when the motion state changes, e.g., by a specifiedamount. When the remote controller 30 is attached to the main body 10,the application running on the mobile device 50 may be able to determinethe starting position and orientation of the remote controller 30 inrelation to the main body 10 or mobile device 50. This information maybe used to track the position and orientation of the remote controllerwith greater accuracy. When the motion data from the remote controller30 is used in a simulation that uses a human armature, the motion can becomputationally mapped to the constraints of the human form, thusproviding a method of using the remote controller 30 as a virtual handand gesture device with high accuracy in terms of the relation to theuser's own hand.

FIG. 8c illustrates two lights L1 and L2 that can be used in place offiducial markers and in conjunction with the mobile device's camera 54(not shown) and computer vision algorithms to detect the relativeposition of the remote controller 30 and the mobile devices 50 asdiagramed in FIG. 9d . In some embodiments, a peripheral attachment port36 as illustrated in FIG. 8a may allow for additional extensions to beadded to the remote controller 30. Peripheral attachments may beornamental in nature for the purpose of representing (or indicating) areal world tool to a user, such as a hammer or ax, or may be functional,such as when used as a fiducial marker 60, as shown in FIG. 9 a.

When using a fiducial marker 60, the mobile computing device's camera 54(see FIG. 4a ) may then capture the fiducial marker 60 for use in or byan application on the mobile computing device 50. In this regard, thefiducial marker 60 may feature different patterns 62 on multiple faceswhich may be read via a camera or an infrared detector, for example, toconvey both location (in relative space, based upon size of the marker)and rotational information (based upon the specific marker(s) visibleand their angle) about the controller 30. FIG. 9b shows the same view ofthe remote controller 30 but with a lighted ball 61 in place of thefiducial marker 60. The peripheral attachment port 36 seen in FIG. 8amay be a common jack (e.g., AUX input jack) for interchanging identicalstems 63 of the fiducial marker 60 and lighted ball 61. The maindifference between the use of the lighted ball 61 and the fiducialmarker 60 is the method of detection (e.g., marker based vs. blobbased).

FIG. 9c discloses a first person perspective of the same fiducial markerfrom FIG. 9a interposed (through augmented reality software) with ahammer head. Using the fiducial markers, the MHMD can combine virtualand real objects from a user's perspective such that “swinging” thecontroller (marked with the fiducial markers) appears as though the useris swinging a hammer. This may be used to provide interactive elementsto a game or augmented reality environment.

This process is illustrated in FIG. 9d . After beginning, the computervision marker detection process 102 is used to search for and, ifpresent, to detect fiducial markers. If a marker is not detected at 104,then the process ends (by beginning again in search of the next marker).

If a marker is detected, then the marker's position and rotation aredetected at 106. Because each face of the fiducial marker (each, amarker in themselves) is distinct, the computer vision software candetermine the distance (relative position to the MHMD camera) and, thusthe location in free space, and the rotation based upon the angle of themarkers presented to the camera.

Next, the visualization engine (e.g. virtual reality or augmented realtysoftware) provides a real-time stream of data (either game data for VRapplications or a video captured by the MHMD camera for augmentedreality) to the wearer with a “virtual” item interspersed within thatdata as oriented, located, and rotated by the user based upon thefiducial marker data observed.

The lens assembly 20 as illustrated in FIG. 6 is one exemplaryembodiment of the lens assembly; however more complicated assembliesthat allow for adjustments of the individual lens positions such asillustrated in FIG. 10a are also contemplated. FIG. 10a illustrates alens assembly 20 with two lens assembly horizontal adjustment pieces 25a and 25 b with interlocking ridges, shown in FIG. 10b as elements 25 a1 and 25 b 1. The two lens assembly horizontal adjustment pieces 25 aand 25 b fit into the lens assembly frame 28 and, as illustrated in FIG.10c , may interlock with the lens assembly frames interlocking ridges 28g, allowing for horizontal adjustment of the lens assembly horizontaladjustment pieces 25 a and 25 b and secure fit. It is also envisionedthat in some embodiments, the lens assembly frame mechanics may beformed out of the foam body 10 without the need of a separate lensassembly frame 28.

FIGS. 10a and 10b shows an exemplary embodiment in which the lens eyepieces 26 a and 26 b screw into the horizontal adjustment pieces 25 aand 25 b to allow rotational adjustment on the z axis. FIG. 10c showsone embodiment of lens styluses 29 a and 29 b with conductive materialCM on the tips. FIG. 10d illustrates exemplary use of the styluses 29 aand 29 b along with the conductive material CM to create correspondingcontact points CPa and CPb on the mobile device screen 52. FIG. 10eillustrates the relationship of the contact points CP and the lensfulcrum.

FIG. 10f describes the process of determining the contact points CPa andCPb and computing any changes in software based on the positions,according to one embodiment. The points CPa and CPb may be fixed, basedupon the design of the lens assembly, such that when the styluses 29 aand 29 b touch the mobile device screen 52, virtual reality, augmentedreality or, more basically, VR or virtual reality driver software mayderive the interpupillary distance between the two eyes. As mentionedabove, the interpupillary distance is useful for properly presentingvirtual reality or augmented reality environments to a wearer of theMHMD.

Because the distance from CPa or CPb to the center of each respectivelens 21 a and 21 b is known, the IPD may be derived therefrom. Theconductive material, thus, provides a contact point with substantialaccuracy (e.g. typing on a capacitive mobile device screen) to enablethe mobile device screen 52 to be adequately calibrated based upon theIPD derived therefrom.

Here, as shown in FIG. 10f , the process begins with detection by thetouch screen of (x,y) coordinate positions of the stylus on the mobiledevice screen 52 at 1001. Capacitive touchscreens typical in most modernmobile devices are capable of simultaneous detection of multipletouches, so this process may take place once for each lens 21 a, 21 b,or may take place simultaneously for both.

If the position is unchanged from the last known position (or abeginning default position) at 1002, then the process returns to thebeginning to await a change. If the position is changed at 1002, then anew lens position is calculated at 1003 based upon the known distance(and angle) of the center of the respective lens 21 a, 21 b, and the(x,y) location of the stylus.

Finally, the virtual reality or augmented reality software (or driver)re-computes any changes to the data displayed on the mobile computingdevice screen 52 at 1004. This may mean that the images shown on themobile computing device screen 52 should be shown further apart orcloser together or with a larger “black” or “darkened” gap between thetwo images in order to ensure that the images presented properlyconverge to a user wearing the MHMD given the updated (IPD). Failure todo so may make a wearer cross-eyed, give a wearer headaches, cause awearer to feel dizzy, or otherwise degrade the experience of the MHMDwearer.

The capability to dynamically detect these positions is necessary in thepresent application because there is no standardized hardware (or IPD)being employed. In situations in which a single screen size is used forall software (i.e. the Oculus VR, Inc., Oculus Rift headset) then theIPD may be pre-set (as it was in the first version of the RIFT)regardless of the wearer. Without adjusting for IPD, the focal point ofthe wearer may be incorrectly calibrated relative to the images beingdisplayed.

Here, in a situation in which the lenses 21 a and 21 b are moveable forthe comfort of the wearer, determining the IPD is an important part ofproviding a quality experience to the user. The introduction of variablescreen sizes, because many different types and sizes of mobile devicesmay be used in the present MHMD, only complicates things further.

Other methods for calculating IPD may also be employed including,incorporating a set of “wheels” or “gears” to enable the lenses to bedynamically moved by a wearer, while set within an MHMD, whilesimultaneously tracking the specific rotation of those wheels or gearssuch that IPD may derived from the current orientation of the wheels orgears. Similarly, a backwards-facing camera (including one built into amobile device 50 that faces the same direction as the mobile computingdevice screen 52 may be capable, in conjunction with suitable software,of detecting the location of one or both lenses 21 a, 21 b based uponfiducial markers, visual markers or other elements interposed on theface of any lens assembly 20.

Turning to FIG. 11a , the main body 10 may be printed or formed with avisual pattern that allows the main body to be identified as a fiducialmarker, as shown in elements 19 a-19 e. The use of a printed pattern isa preferred method; however, other methods and means that allow forcomputer vision detection, such as the use of decals, or a 3D softwarerepresentation (e.g., model) of the main body 10 or any component of thesystem or the system's physical form as a whole, are also contemplated.Exemplary methods of use of patterns on the main body 10 will bedescribed below with reference to FIGS. 11b -11 d.

Preferably, the main body 10 may be entirely or primarily formed from adurable foam material. This material provides flexibility, especially toflex inward for smaller heads and spread apart for larger heads, asillustrated in FIG. 12, and may be light-weight compared to typicalsolid construction materials, e.g., plastics or metals. The material mayallow for a snug fit for a large range of head sizes, providing aone-size-fits-all solution. In addition, the durable foam may alsoprovide for comfort as it is worn by the user by allowing the main body10 to adapt to the facial shape of the user and distribute pressurecaused by the weight of the system. Further, the density of the materialmay allow for stability of the overall structure and the variouscomponents. That is, the foam unibody 10 has the ability to absorbimpacts, torsional and compressive forces that might be harmful tosomething with a rigid structure. Indeed, the mass of the unibody 10adds suitable rigidity. Also, the use of a foam material may allow for asimplified construction process (manufacture) as compared toconstructions that use hard structural frames for support in addition toa soft material for comfort, e.g., a foam pad interposed between a hardstructural frame and the user's face/head. The foam material can beformulated with anti-microbial chemicals, which may provide betterhygiene than other materials. The use of closed cell foam or any foamwith a (e.g., non-permeable) skin permits easy cleaning and thusprovides additional hygienic benefits in comparison to other materials.

The use of foam material to construct the main body (and/or otherportions of the apparatus) may allow one or more of the componentsdescribed above to be omitted or replaced, where the foam materialitself provides the functionality of the omitted components. Saidanother way, the foam construction may provide the functionalitydescribed above with respect to one or more of these components, and sothe component as a separate piece of the apparatus may be omitted. Saidin yet another way, the components and/or their functionality may beimplemented by the foam material construction, e.g., of the main body,thus rendering the use of separate and distinct components for thesefunctions unnecessary.

For example, the use of foam material allows for the omission orreplacement of (separate) external frame 19 as described in FIG. 4b .That is, the foam material as part of the main body 10 is constructed tosecure and position the mobile computing device 50 as described in FIG.4b either with or without the internal frame.

As another example, the use of foam material allows for the omission orreplacement of (separate) components 51 a, 51 b and 51 c of the internalframe 51, as described in FIG. 4c . In other words, the foam material aspart of the main body 10 may be constructed to secure and position themobile computing device 50, as described in FIG. 4 c.

As yet a further example, the use of foam material allows for theomission or replacement of (separate) components of the lens frame 28,as described in FIG. 10c . In other words, the foam material (as part ofthe main body 10) may be constructed in such a way as to provide thefunctionality of the components 28 and 28 g described in FIG. 10c asfeatures of the main body 10, i.e., providing equivalent functionalcapabilities to allow horizontal adjustments of the lens assemblyhorizontal adjustment pieces 25 a and 25 b and secure fit.

The main body 10 may have a unibody construction, i.e., the main bodymay be a single piece of foam material.

Note that other materials such as rubber, plastic, or combination ofmaterials and structure such as an interior frame wrapped with lessdense foam covered in a fabric mesh, may also be used as desired.

Exemplary Method of Use

The user 70 may run (execute) a system compatible application on themobile computing device 50. In some embodiments, once the applicationhas loaded and following any set-up steps required by the application,the user may insert the mobile computing device 50 into the slot 18 ofthe main body 10, or into the mobile computing device frame 19 and theninto the slot 18 of the main body, or otherwise incorporate the mobilecomputing device into the system. The user may then affix the system tohis/her head by positioning the main body 10 in front of their eyes,much like wearing a pair of goggles or glasses. The user may thenposition the strap 40 around their head so that the main body 10 issecured to the user's head. The user may now see the mobile computingdevice 50 (or more specifically, the screen thereof) through the lensassembly 20, where the lens assembly may allow each eye to see only adiscrete (respective) portion of the mobile computing device screen 52,which allows for a 3D or stereoscopic viewing experience. Alternatively,the user may don the main body, then insert or attach the mobilecomputing device.

Depending on the application, the user may use the remote controller 30to interact with the application via controller motion and/or buttonpresses. The remote controller 30 may send information to the mobilecomputing device 50, which may expose (or communicate) the informationto the (system compatible) application, where the information may beprogrammatically used to interact with the application. The types ofapplications envisioned include augmented reality, virtual reality, and3D media type applications; however the use of the system for othertypes of applications is contemplated and expected, and dependent on theapplication.

For example, in one exemplary case of a virtual reality application, theuser may be (virtually) placed in a virtual environment where theapplication may display a stereoscopic image of the virtual environmentonto the mobile computing device screen 52. In the case where the mobilecomputing device contains motion sensors, the movement of the device maybe interpreted in the virtual world as controlling a virtual cameramimicking or tracking the motion of the user's head. This may allow theuser to see into the virtual world and look around as if the user wereactually there.

In cases of computer vision applications, the device camera 54 may beused to identify fiducial markers. For example, the application runningon the device may utilize computer vision to “see” (and recognize) afiducial marker of or on a viewed item in the camera video feed. Once afiducial marker is detected, a virtual object may be displayed on top of(or overlaid on) the stereoscopic video, to the effect that the virtualobject is presented in the real world at scale, rotation, and position,relative to the user. The user may then interact with the object withthe remote controller 30 or through movement.

The user may fit the remote controller 30 with a fiducial marker 60 toallow detection of the remote controller in the camera field of view(FOV). FIG. 9b shows an exemplary attachment of a virtual object VR4, inthis case a hammer, to the remote controller 30, where the virtualobject appears in the rendered 3D scene (but isn't actually present inthe real world).

The main body 10 may be used as, or configured with, a fiducial marker.FIG. 11a illustrates an embodiment in which the sides of the main body10 are designed or provided with individual textures 19 a-19 e that mayact or function as a cubical marker allowing the main body to bedetected from multiple angles. As indicated in FIG. 11b , when usersview instances of the MHMD via respective views V1 and V2, e.g., from aseparate device capable of detecting the textures, the main bodytextures may be used to place virtual objects on or near the main MHMD(instance) in virtual or augmented space, as illustrated in FIG. 11c byrespective views VR1 and VR2. More particularly, virtual dinosaur (VR1)and rabbit (VR2) masks are generated by the processor within the mobilecomputing device 50 in one MHMD and placed over the other MHMD as apresumed object. The ability to track the exterior surfaces of therespective bodies 10 enables each MHMD to move the respective masks inthe same manner that the corresponding person moves their head. In onemethod, summarized in the flowchart of FIG. 11d , a computer attempts tovisually detect a marker in step 102. If a marker is detected in step104, its position and rotation are determined in step 106. Finally, avisualization engine is used to show the virtual item in step 108.

In some embodiments, toys or other physical objects may be used asmarkers. FIG. 14a shows the user looking at a physical object 80, inthis case a dollhouse. In one exemplary embodiment, the computer visionalgorithms running in the application may be pre-programmed with thephysical object 80 shape, and virtual objects VR5-VR8 may then bepositioned and interact with a 3D representation of the object, asindicated in FIG. 14b . These virtual objects VR5-VR8 can be placedaccurately into the device video feed, merging the virtual and physicalspaces known as augmented reality, as illustrated in FIG. 14b . The usermay then use the various features of the system to interact with theaugmented reality experience. Furthermore, given the pliable nature ofthe foam material as well as its durability and ability to protect themobile computing device, the headset is well-suited to be used in a playenvironment along with other toy items, even those that might beaccidentally collide with the goggles. This is a distinct advantage overprior MHMD goggles for more active users since the rigid plastic casesmay break or not provide adequate cushioning for the user's head.

FIG. 14c is a flow diagram of an exemplary method 300 for detection of aphysical object and placement of a virtual object, according to oneembodiment. As FIG. 14c shows, once the method begins, and a knownobject is detected in step 302 via computer vision (by the applicationrunning on the mobile computing system), if a (fiducial) marker isdetected in step 303, the method may determine the marker's position androtation (orientation) in step 304, and in step 305 a correspondingvirtual item may be displayed or shown by a virtualization engine, e.g.,of the application.

Computer vision algorithms running on or in the application may make useof point clouds or natural features detection to determine the position,location, and/or size of objects in the physical world, and the user maymove or position themselves relative to these objects.

FIGS. 15a-15c are directed to use of point clouds. More specifically,FIG. 15a is a perspective view illustrating use of point clouds todetermine user perspective and scale of physical environments, accordingto one embodiment. FIG. 15a shows exemplary point cloud data P beingcaptured from the mobile computing device camera, and the resulting orcorresponding 3D space. If the point cloud data P matches a previouspoint cloud data the application may display predetermined virtualcontent.

FIG. 15b is a perspective view illustrating a virtual environment placedonto a physical environment based on point cloud data, according to oneembodiment. As may be seen, FIG. 15b shows a virtual world orenvironment VR9 that fits on top of (or is overlaid on) real worldobjects. This may allow the user to move around the space in a virtualworld where, by avoiding objects in the virtual world, the user alsoavoids objects in the physical world. In some embodiments, the systemmay allow dynamic generation of virtual content based on the point clouddata or natural features detection. For example, the application mayinclude dynamic scene generating algorithms that use the point clouddata or natural features detection algorithms to determine the physicalspace and using the computed space to place virtual objects that overlayonto the physical world. One exemplary process for doing so is outlinedin FIG. 15c .

FIG. 15c is a flow diagram of a method for using point cloud data todisplay a virtual environment, according to one embodiment. As shown, inone embodiment, in step 401 a point cloud may be detected via computervision. In step 402, if the point cloud is identified, i.e., is a knownpoint cloud, a virtual space may be determined based on the point cloudposition and rotation (or orientation), as indicated in step 403, and avirtualization engine (e.g., included in the application) may show avirtual item accordingly, as indicated in step 404.

If, on the other hand, the point cloud is not known (in step 402), thenas indicated in step 405, if dynamic object creation is not implementedor enabled, the method may return to the beginning, as shown.Alternatively, if dynamic object creation is implemented or enabled,then in step 406 corresponding physical objects may be determined, andvirtual objects matching the real (physical) objects may be dynamicallygenerated, as indicated in step 407.

Radio signals may be used for relative or absolute positioning amongMHMDs. FIGS. 16a-16c are directed to the use of a signal processingserver that may be used to implement this functionality. FIG. 16a is aperspective view illustrating interaction between mobile computingdevices and a signal processing server 110, and FIG. 16b is a top viewillustrating interaction between the mobile computing devices and signalprocessing server 110. The use of a signal processing server 110 asshown in FIG. 16a and FIG. 16b may allow positional tracking of multipleusers, labeled POS0, POS2, and POS3 in the Figures. The mobile computingdevice may add orientation data to the position data to get an accuratelocation, orientation, and movement of the user or multiple users invirtual space (VS) and in real world space. If the location of thesignal processing server 110 has a known position in 3D space the user'sposition may be determined, and a virtual world or virtual object may beplaced accurately with respect to the user. If the position of thesignal processing server is not known, the position of the user may beknown (or determined) relative to the signal processing server and toother users, but possibly not in the real world space. In someembodiments, the mobile computing device may operate as a signalprocessing server, or multiple devices may be used to determine relativeor absolute positioning. The locations of multiple users and the signalprocessing server 110 may be shared in a multiplayer experience allowingmovement, interaction, and manipulation of the virtual space together.It is contemplated that any use of positional data used by the systemmay also be used in a multiple user scenario where location data ofenvironmental feature locations and/or user positional and orientationdata may be shared via a network. The result provides, by way of usingadditional sensors and systems, a more robust global spatial awarenessthat can be shared among the individual users.

FIG. 16c is a flow diagram 450 of a method for interaction betweenmobile computing devices and a signal processing server to determinerelative position and orientation. As FIG. 16c shows, in one embodiment,in step 451, a device signal may be received by a signal processor. Instep 452, a position may be calculated, e.g., in x, y, z,space/coordinates, and in step 453, the position may be sent to thedevice, which may determine an orientation, as indicated in step 454,and may send the orientation to a server, as per step 455. Additionally,after the orientation is determined in step 454, the device may requestone or more other devices' positions and orientations from the server,as indicated in step 456.

In the use case of viewing 3D media, the user may load media content oran application that displays the media in a side by side format (e.g.,in a stereoscopic format). The user may then view the media through thelens assembly 20 and may optionally use headphones 80, thus creating a3D media experience.

Additionally, many more experiences are contemplated that do not fallunder one of the above categories. The use of the mobile computingdevice 50 features not mentioned herein and many features that may beavailable in future mobile computing devices may enable developers tocreate applications and experiences for the system that are not listedabove.

Note that the remote controller 30 illustrated in FIG. 7a is but oneembodiment, and numerous other configurations of the remote controllerare contemplated that may include and utilize additional buttons andtriggers and additional re-attachment methods, as indicated in FIG. 7band FIG. 8b . Note, for example, that while in FIGS. 1, 2, and 3 a, theremote controller is attached to the MHMD or the remote controller maybe held in the user's hand.

FIG. 13a and FIG. 13b show an alternative example of the main bodyintegrated with electronic components. In this example, the componentsillustrated are heart rate monitors 91 a and 91 b, EEG sensors 90 a, 90b, and 90 c, stereo speakers 92 a and 92 b, and a circuit board withmicrocontrollers 96 and wiring 99. The mobile computing device 50 may beused to interface with the various components via a devices data input,audio port, or wireless communication, as desired. The electroniccomponents may receive power from a battery (not shown) integrated intothe main body 10, or by using power from the mobile computing device 10.

As an alternative, FIG. 13a shows an alternative in which the mobilecomputing device 50 is inserted into slot 18 on the side of the mainbody 10. Other ways of inserting or attaching the mobile computingdevice 50 to or in the main body may include separate pieces ofconstruction of the main body that allow for mobile computing deviceswith a range of sizes and form factors to be inserted and secured intothe main body. And as described previously, FIGS. 4b and 4c illustratethe use of frames 19 and 51 to secure the mobile computing device 50 forinserting into a slot 18 of the main body 10.

FIGS. 17A and 17B show another example of a mobile head mounted display(MHMD) goggles 500 from the front and rear. As described above, thegoggles 500 comprise a soft main body 502 having a generally rectangularprism shape on its front side and a concave face-contacting lip 504 onits rear side. A pair of adjustable lens assemblies 506 a, 506 b eachhaving a lens 507 (FIG. 19) are mounted within a hollow interior cavity508 of the goggles 500. An elongated vertical pocket 510 opens upward inthe body 502 to permit introduction and retention of a mobile computingdevice (not shown) into the cavity 508. The pocket is shown as enclosedon all sides but one, but may be open from the bottom, from the back,from the face the side or other locations. Various openings, both forinsertion of and for securely holding a smartphone in the pocket areenvisioned. As will be described, the display screen of the mobilecomputing device faces to the rear, directly in front of the lensassemblies 506 a, 506 b. One or more remote controls 512 may beremovably secured to the main body 502 for use in conjunction with themobile computing device. Further details on the advantages of theseremote controls 512 will be explained below.

As mentioned above, the type of mobile computing device may varydepending on the size of the vertical pocket 510. For example, pocket510 may accommodate modern smartphones or maybe larger to accommodatetablet computers. The term “smartphone” will be used hereafter in placeof “mobile computing device.”

As described previously, the goggles 500 are preferably retained on aperson's head using retention straps. For example, a rear strap 514extends around the backside of a wearer's head, as seen in FIGS. 18A and18B. An overhead strap 516 also may be provided to help prevent thegoggles 500 from slipping down the user's face. Each of the straps 514,516 are secured to grommets or reinforced inserts 517 that closely fitwithin channels on the sides and top of the main body 502, and arepreferably adjustable for different sizes of heads. FIG. 19 shows anupper grommet 517 and two side grommets 517 exploded from the main body502, each of which may be secured to the main body via adhesive or asimple interference fit. The grommets 517 are formed of a more rigidmaterial than the body 502 to withstand the greater tensile forcesapplied thereto.

FIG. 18A shows a person wearing the MHMD goggles 500 in a first mode ofoperation wherein the remote controls 512 are docked on the sides of themain body 502. In this mode, the user can still manipulate controlbuttons on the outer face of the remote controls 512 while viewingcontent displayed on the smartphone.

In a second mode of operation, seen in FIG. 18B, the user has removedone of the remote controls 512 and is holding it in his or her hand. Oneor both of the remote controls 512 can be “undocked” in this manner andused in various contexts, as has been explained above and will bedescribed further below.

With reference back to the perspective views of FIGS. 17A and 17B, andalso the exploded view of FIG. 19, the remote controls 512 desirablyattached to side walls 518 of the main body 502 using docking clips 520.In FIG. 19, the two remote controls 512 are shown exploded to eitherside of the main body 502 along with the docking clips 520. Each of thedocking clips 520 has a central neck portion 522 in between an outerclip portion 524 and an inner anchor portion 526, both of which areenlarged with respect to the neck portion.

Because of the softness and pliability of the material of the main body502, the inner anchor portions 526 of each of the docking clips 520 canbe pushed through vertical slots 528 formed in the side walls 518 untilthe anchor portions are past the slots and within the interior cavity508 of the main body 502. That is, the narrow neck portion 522 has ahorizontal length that is substantially the same as the thickness of theside walls 518 such that the clips 520 are held firmly with respect tothe main body 502. This is seen best in the horizontal section view ofFIG. 21B. Although not shown in great detail, the outer clip portion 524include attachment structure 525 that mates with correspondingattachment structure 527 provided on the bottom faces of the remotecontrols 512 (see also FIG. 26B). The mating attachment structurespermit easy docking and undocking of the remote controls 512. Forexample, the attachment structure 525 on the clips 520 may be T-shapedso as to slide into and be captured by slots 527 that include a largeentry opening and smaller retention segment. In this way the controllers512 are simply slid on and off of the sides of the goggles 500, and heldby friction.

The docking clips 520 may be clips of another form entirely or may useother attachment structures. For example, in place of the docking clips520 Velcro®, adhesive pads, locking mechanisms, latches, grommets,magnets and other, similar, attachment structures may be used. The useof docking clips 520 is only the preferred option. Still further, aconcave depression shaped like the back face of the remote control 512may be formed in one or both side walls 518 of the main body so as toclosely receive the remote and reduce its outward profile extendingoutside of the soft body. This latter solution helps reduce movement ofthe remote control 512 relative to the main body, thus reducing thechance of detachment from head movement.

FIG. 17B illustrates a generally flat vertical front wall 530 of themain body 502 having a window 532 on its right side (the directions leftand right being as perceived by a wearer of the goggles 500). Asmentioned, a smartphone may be inserted into the vertical pocket 510 sothat its display screen is visible in the interior of the goggles 500.Many such devices have rear facing camera lenses, and thus the window532 provides an opening for these lenses. Accordingly, a wearer of thegoggles 500 can initiate real-time video through the smartphone to beseen on the internal display, for use in an augmented reality (AR)program, for example.

FIGS. 20A-20L are various orthogonal and sectional views of a soft mainbody 502 of the MHMD goggles 500. The main body 502 has a shape and ismade of a soft material so as to result in a number of distinctadvantages over prior MHMD goggles. Primarily, the main body 502 is madeof a soft foam which will flex to fit different shapes and sizes offace, making it easier to fit universally, and more comfortable in theprocess. The softness of the main body 502 an “approachable” aesthetic,which is important to inducing people to put such an HMD on their facein the first place. Indeed, the soft foam permits the entire main body502 to be compressed down to a very small profile. The use of thesegoggles 500 in environments such as public arcades and other placeswhere the goggles may be loaned or rented out means that their ergonomicqualities are magnified. That is, if the general public perceives thegoggles as comfortable and easy to move around in, they are more likelyto pay a return visit and share their experience with others. Moreover,by inserting one's smartphone into the vertical retention pocket 510 issurrounded by a soft, cushion-like material of the main body 502 whichprovides significant shock-absorbing protection if the goggles aredropped, for example.

In this regard, the soft main body 502 is a comfortable “face feel”making it more tolerable to wear the goggles 500 for a longer period oftime and enabling the entire main body 502 to conform around a wearer'sface. Furthermore, a preferred foam material makes the main body 502extremely light weight, and the weight of the other components such asthe lens assemblies 506 and remotes 512 are kept down so that thegoggles 500 are easy to wear for long periods of time. Preferably, thegoggles 500 have a maximum weight of about 150-230 gf with the headstrap and lenses (but without the remotes 512), though certain foamformulations may reduce that further.

The material of the soft main body 502 is preferably a soft flexiblefoam, more preferably a closed-cell foam or a so-called “Integral Skin”foam. The formulation of the foam material may vary, and includesEthylene-vinyl acetate (EVA), Polyurethane (PU), and HE foam. Each ofthese alone or in various combinations may be utilized. It should beunderstood, however, that any material that can be molded into the shapeof the main body 502 may be used, and though foam is preferred it is notthe exclusive option. The main preference is the ability to mold thematerial into shape such that when it is molding is complete, thematerial is soft, impermeable, and compressible. In addition, thematerial may be soft to the touch, and because the entire main body 502is formed of the material, the entire main body 502 is soft to thetouch. The material may have a relatively high tensile strength toresist wear and tearing. Some prior head mounted goggles utilizeseparate pieces of injection-molded plastic coupled together which arebrittle and, as a result, tend to break at the seams/junctions.

In a preferred embodiment, the entire main body 502 is formed of asingle, homogeneous unitary foam member which may be injection molded,pour molded, or cold-form molded. The advantages of having a singleunitary foam member include low manufacturing cost because there is onlya single mold and no assembly of components required, and structuralintegrity because there is less opportunity for breakage at joints orseems between multiple different parts. The molded foam manufacturingtechnique accommodates complex internal shapes (e.g., slots for lensassemblies, nose bridge), and permits the inclusion of ancillary partssuch as the strap anchors, either by being molded into the goggles orwith the provision of shaped recesses and the like. Molding permits theinterior walls to provide an appealing “face feel” and any desiredtexturing (to aid in grip of the face as well as comfort). The use of afoam “hunibody” also allows for distinct outer shapes to be easilyproduced without affecting the mechanical functionality of the main body502, thus allowing custom physical designs of the goggles that have adistinct look and feel to be easily manufactured. Finally, multiplecolors and designs may easily be incorporated into the foam, includingbranding or advertising on any of the generally flat outer surfaces ofthe main body 502.

Alternatively, the main body 502 may be formed of an inner structural“skeleton” of sorts covered by a molded soft foam. In this embodiment,an internal portion or skeleton of the main body 502 is first moldedwith a higher density foam, or other plastic, and then the variousinternal and external contours of the main body 502 are formed bymolding the softer foam around the skeleton. Although there areessentially two components of this type of body 502, because they aremolded together into one piece they may also be referred to as a unitaryfoam member. In other words, once molded there is no need for attachingpieces together to form the body 502. Still further, the aforementionedinternal frames 50, 51 or other internal components may be formed byinserts of material that is less compressible than the softer foam. Forinstance, inserts or frames may be combined with a soft foam body todefine the retention pocket 510 or channels within which the lensassemblies 506 slide.

Furthermore, the use of a closed-cell or other water-resistant foampromotes hygiene and permits the main body 502 to be easily cleaned.That is, ancillary components such as the lens assemblies 506 and theremote controls 512 may be removed and a water-resistant foam body 502may be wiped down or even immersed in water for cleaning. Foam typesthat are water-resistant, at least more so than open cell foams, includeclosed cell foams and Integral Skin foams. The latter includes an outersubstantially non-porous skin formed during the mold process against themold surface. Other materials that have been used are incapable of beingeasily disassembled or tend to absorb contaminants, whereas theclosed-cell foam provides an exterior barrier to such contamination. Ina further embodiment, the material may be seeded or coated with anantimicrobial chemical to kill bacteria.

With reference to FIGS. 20A-20E, the various contours of the main body502 are illustrated in greater detail. As mentioned, the front portionof the main body 502 has a generally rectangular or box shape, while therear portion has a contoured lip 504 which fits the user's face. Theside walls 518 may be generally perpendicular to the front face 530, orthey be slightly tapered inward in a rearward direction. The side walls518 terminate at a pair of temple contact members 534 whose rear edgesform a part of the contoured lip 504. The lip 504 further includes anupper edge 536 for contacting the forehead of the user, a lower edge 538that contacts the user's cheeks, and a nose bridge 540. The contouredlip 504 resembles the same features as on an underwater scuba mask, andin contacting and conforming to the face of the wearer prevents lightfrom entering the interior cavity 508 from the rear. As was mentionedabove with respect to FIG. 12, the temple contact members 534 flex inand out to fit various sized heads by virtue of the soft foam material.The rear straps 514 (FIG. 17B) preferably attach to anchor pins 542recessed on the outside of the side walls 518, just in front of thetemple contact members 534. In this way, the straps 514 can easily pullthe side walls inward into contact with a smaller head. The combinationof shape and material conform well to a very wide array of facialdimensions and the relatively large interior cavity 508 and ability toflex accommodates people wearing glasses. Alternatively, indents on theinner walls may be molded in to provide reliefs for eyeglass stems. Thefoam material absorbs movement and vibration and tends to provide asecure “anchoring” effect to keep the goggles 500 in place during headmovements.

Now with reference to FIGS. 20E and 20F, advantageous retention andpositioning features within the vertical pocket 510 will be described.Angled lines 20F-20F shown in FIG. 20E extend across the pocket 510looking forward so that the features on the inside of the front wall 530are shown in FIG. 20F. In particular, the soft foam of the main body 502is molded to induce automatic or passive leveling and centering of thesmartphone as it is being inserted into the pocket 510, regardless ofsize. The width of the pocket 510 may vary depending on the type andsize of mobile computing device for which the goggles 500 are designed,though, as mentioned, to keep the overall size of the goggles down theyare typically meant to hold and retain a smartphone. The average screensize for smartphones in 2015 is about 5 inches (12.7 cm), meaning anoverall length of phone of just under 5 inches. For instance, the iPhone6 has a screen size of 4.7 inches (11.9 cm), while the iPhone 6 Plus hasa screen size of 5.5 inches (14.0 cm), and the trend is for even largerphones. An exemplary width of the vertical pocket 510 is about 5.7inches (14.5 cm), although as mentioned above larger goggles toaccommodate larger smartphones or even tablets are contemplated. Anotheradvantage of the foam material is that the pocket 510 may stretch toaccommodate phones that are slightly larger than the pocket for whichthe phone is originally designed.

The rear face of the front wall 530 is generally flat and vertical, butincludes a pair of relatively large ramped protrusions 544 projectingrearward from into the pocket 510. These protrusions 544 are locatedtoward the top of the pocket 510 and are largest on their outer extentsso as to contact and force both ends of the smartphone inward. That is,if the device is inserted off-center, the protrusions 544 tend to centerthe device. Furthermore, a plurality of smaller friction bumpers or nubs546 also project rearward from the front wall 530 into the pocket 510.These nubs 546 are generally evenly distributed in two rows at the topand the bottom of the slot, as seen in FIG. 20F, so as to applysymmetric compression forces against the smartphone and hold it in anorientation which is perpendicular to a front-rear horizontal axisthrough the body 502.

The smartphone inserts in the pocket 510 between the rear face of thefront wall 530 and in front of an internal divider wall 548 that extendsparallel to the front wall, and is seen best in FIGS. 20B and 20K. Thedivider wall 548 is not a slab, but instead includes two identicalrelatively large apertures 550 separated by a central partition 552through which the lenses 507 of the lens assemblies 506 visualize thedisplay screen of the smartphone. The divider wall 548 provides aperipheral frame oriented in a vertical plane against which abuts thefront edges or bezel of the smartphone. The horizontal distance betweenthe nubs 546 and the divider wall 548 is desirably size less than theminimum thickness of the smartphone expected to be inserted therein suchthat the foam nubs 546, and the divider wall 542 to a certain extent,are compressed when the device is inserted. Of course, the rampedprotrusions 544 being larger than the nubs 546 are compressed againstthe rear face of the smartphone even more. The compression of the foamsurfaces on both faces of the smartphone securely retains it within thevertical pocket 510.

As an additional precaution to retain the smartphone within the pocket510, a pair of inward ledges 554 are formed at the top end of the slot,as seen in FIG. 20L. These ledges 554 even overlap to a certain extentto prevent the phone from falling out when the HMD is held upside down.

FIG. 20G shows an alternative arrangement of the leveling and centeringprotrusions 556 extending inward into the pocket 510. Rather than beingon the front wall 530, the protrusions 556 extend from each side wall518. Since these protrusions 556 require side wall support, two smallslots 558 provide access to the ends of a smartphone 572 placed withinthe pocket 510 for connection of audio jacks, power cords, etc.Insertion of the smartphone 572 as seen in FIG. 20H compresses the sideprotrusions 556 which, in turn, apply approximately equal inward forceon the smartphone so that it is laterally centered in the pocket 510.Although not shown, similar protrusions or bumpers may be provided atthe bottom of the slot for horizontal leveling. The friction bumpers ornubs 546 as shown in FIG. 20F are also present to maintain the phoneperpendicular in the body 510.

FIG. 20E shows a top wall 560 of the main body 502, while FIG. 20Iillustrates a bottom wall 562. Lens adjustment slots 564 a, 564 b areformed in both the top wall 560 and bottom wall 562. More particularly,a pair of vertically aligned left-side lens adjustment slots 564 a areformed, one in the top wall 560 and one in the bottom wall, and a pairof vertically aligned right-side lens adjustment slots 564 b are formed,one in the top wall 560 and one in the bottom wall. These slots 564received and permit lateral adjustment the lens assemblies 506 a, 506 b,as will be described below. Both the top wall 560 and the bottom wall562 each include a pair of vent holes 566 that are positioned betweenthe slots 564 and the face-engaging lip 504 so as to help reducehumidity and fogging of the lenses 507 within the goggles 500. FIG. 201further illustrates a narrow aperture 568 formed in the center anddirectly below the vertical retention pocket 510. This aperture 568enables the user to easily push the smartphone from below out of theretention pocket 510.

FIG. 20J again shows the side wall 518 of the main body 502 having thevertical slots 528 for receiving the docking clips 520 to hold theremote controllers 512. In addition, relatively large vertical slots 570are provided in both side walls 518 opening to the pockets 510. Thevertical slots 570 provide access to the ends of the smartphone withinthe pocket 510 for connection of audio jacks, power cords, etc.

FIG. 21A is a side elevation view of the MHMD goggles 500, and FIG. 21Bis a horizontal sectional view through the goggles showing a smartphone572 positioned within the vertical retention pocket 510. FIG. 21B alsoillustrates the relative position of the two remote controllers 512 whenthey are docked. Once again, the somewhat I-beam shaped docking clips520 are held within the slots 528 (FIG. 20J) in the side walls 518, andsecure the remote controllers 512 in an easily detachable manner.Desirably, small bumps 574 extending outward from both side walls 518just forward of the slots 528 contact switches 576 (FIG. 26B) on theback of each remote controller 512 to signify when the controllers areproperly docked. In this manner, the precise position of the controllers512 is calibrated whenever they are docked to the sides of the goggles500. A more complete explanation of the capabilities of the entire MHMDgoggles 500 with the controllers 512 will be provided below with respectto FIGS. 26-27.

FIG. 21B best shows the positions of the lens assemblies 506 a, 506 bwithin the goggles 500, and FIGS. 22A and 22B are front and rearperspective views of the lens assemblies. In contrast to the lensassembly 20 described above with respect to FIGS. 10a-10c , the left andright lens assemblies 506 a, 506 b are completely separate and do notshare a common frame. Each of the lens assemblies 506 a, 506 b is shownwithout the actual lenses 507 in these views to provide greatervisibility of the various components within the goggles 500. In apreferred embodiment, the lenses slightly magnify the field of view andtheir focus may be adjusted by rotating the lenses within circularbezels 580. The bezels 580 project to the rear from an outwardlyrectangular frame 582 which has upper and lower posts 584 terminating infinger pads 586.

As seen in FIG. 17A, the lens assemblies 506 a, 506 b are positionedwithin the main body 502 of the goggles such that the rectangular frame582 is oriented vertically and positioned just in front of thesmartphone retention pocket 510. FIG. 20K illustrates inner channels 590formed by the main body 502 including small guide walls 592 that closelysurround the rectangular frames 582. The lateral width of the channels590 is greater than the width of the rectangle frames 582 such that thelens assemblies 506 a, 506 b can be moved side to side. The upper andlower posts 584 are somewhat blade-like so as to fit closely within theupper and lower lens adjustment slots 564 a, 564 b described above withrespect to FIGS. 20E and 20I. The lateral width of the adjustment slots564 a, 564 b is also greater than the width of the posts 584. Asdiscussed above with reference to FIG. 20B, the lenses may be dividedfrom another by a central partition 552 running substantially up to thesmartphone screen.

FIG. 23 is a view looking down on the main body 502 shown in phantom andillustrating the side-to-side adjustability of the independent lensassemblies 506 a, 506 b. The wearer need only squeeze both upper andlower finger pads 586 to slide the lens assemblies laterally. Theability to adjust the lens assemblies 506 a, 506 b in this manner allowsa user to space them apart in an optimal manner so that the optical axesof the wearer's eyes aligns with the optical axes of the lenses. Easilyadjusting the interpupillary distance (IPD) in this manner allowsdifferent users to comfortably wear the goggles in rapid successionwithout an extensive calibration process.

As was described above, the goggles 500 provide a system for detectingand communicating to the smartphone 572 the individual lens horizontaland vertical positions within the headset. This establishes theinterpupillary distance (IPD). One means for automatically determininginterpupillary distance is to take advantage of the capacitive touchfeatures of the mobile device screen in conjunction with a stylus 594attached to each lens assembly 506. FIGS. 22A and 22B also show anelongated stylus 594 projecting forward from the lens assembly 506. Thestylus 594 preferably terminates in a rounded or bullet-shaped soft tip596 which is designed to contact the display screen of the smartphone572. As seen in both FIGS. 21B and 23, and in a preferred embodiment,the length of the styluses 594 is such that the tips 596 come intocontact with the smartphone 572. FIG. 23 also shows the relative lateralpositions of the styluses 594 to the inside of each lens assembly 506,and as seen in FIG. 22B the stylus is at the bottom of the frame 582, soas to be essentially hidden from the wearer's line of sight—generallyaligned with the wearer's nose. The soft tips 596 are soft polymer orelastomeric while in another embodiment the tips are coated in aconductive paint or may use a conductive foam or any other material thatprovides a capacitive response to the mobile device. Positioningsoftware provided with the goggles 500 may be incorporated into thesmartphone 572 such that when the stylus tips 596 contact the screen ofthe smartphone 572, and the wearer signals that the correct lensposition is reached, the precise position of the optical axis of thelenses within the lens assemblies 506 relative to the smartphone iscommunicated. Alternatively, the stylus tips 596 may constantly be incontact with the screen of the smartphone 572 such that the smartphoneis constantly aware of the location of the lens assemblies 506. Asdiscussed above with respect to FIG. 10f , this location may be used toderive the interpupillary distance (and, indeed, the location of thelenses relative to the screen).

Capacitive touch screens, such as on smartphones, have varyingsensitivities, and a response may be triggered in some from a simpletouch from an inanimate and non-conductive object. A conductive path isnot required if the stylus material conductive properties allow for thetouch response to be triggered. However, this may create a problem withbuildup of charge in the material, and is may be impeded by thedifferent sensitivities of smartphone capacitive screens. Nevertheless,this is considered a viable method of transferring touch inputs withoutthe need of a conductive path. More commonly, an electrical current suchas directly or indirectly from a user's fingertip is necessary, or atleast the use of a stylus with a magnet or some form of ferrous materialin its tip. The present application contemplates styluses integratedwithin the MHMD goggles that transmit a touch and initiate a touchresponse on capacitive touch screens regardless of the means. Thus, theterm “touch input” encompasses all such configurations.

FIGS. 22A and 22B illustrate a button 598 provided in the center of theupper finger pad 586. The button 598 may be configured in severaldifferent ways. In one embodiment, the stylus tips 596 are positioned soas to be slightly away from the screen of the smartphone 572, and thebuttons 598 initiate a mechanical linkage which pushes the stylus tipsagainst the smartphone screen. Two different alternatives this systemare shown in FIGS. 24-25. Alternatively, the buttons 598 may beconstantly in contact with the screen through an electrical circuit tothe tips 596 such that capacitive contact with the styluses 594 with thescreen can be detected based on changes in electrical current. That is,the tips 596 remain in contact with the smartphone screen but anelectrical current from the user's fingers is not transmitted until thebutton 586 is depressed. In either embodiment, when the user merelytouches the lens assembly buttons 598, thereby generating a capacitivechange through the button 598 and conductive stylus 594 to the tips 596and to the screen, the device touch input is registered. In the systemdescribed, two touch inputs provided by the two styluses 594 are used,but it is envisioned that four or more touch inputs could achieved bythe addition of additional styluses and corresponding buttons.

FIGS. 24A-24E illustrate a first alternative lens assembly 600 with amovable stylus tip 602 for use in any of the MHMD goggles describedherein. As before, a bezel 604 mounts within a frame 606 sized to slidelaterally within a main body of the goggles described herein. A pair ofupper and lower finger pads 608 allow a user to displace the lensassembly 600 laterally within the main body, again as described above.The upper finger pad 608 mounts on the end of a pivoting lever 610 whichhas an angled cam surface 612 close to its pivot point (not shown). Thecam surface 612 contacts and acts on a proximal arrow-shaped end 614 ofa shaft 616 positioned to slide axially within the stylus tube 618. Acompression spring 620 positioned within the interior of the stylus tube618 biases the shaft 616 in a proximal direction toward the cam surface612. In this respect, the distal end of the stylus tube 618 is closedexcept for a narrow aperture through which extends a reduced diameterportion of the shaft 616. The stylus tip 602 attaches to a distal end ofthe shaft 616 outside of the stylus tube 618. As seen in FIGS. 24C and24E, when the wearer depresses the finger pad 608, the angled camsurface 612 forces the arrow-shaped shaft end 614 distally whichdisplaces the stylus tip 602 against the smartphone screen. Because of aconductive path extending between the stylus tip 602 and the finger pad608, this registers a touch to the smartphone screen. It should beunderstood that the movable finger pad 608 (or, actuator) could beeither on the top or bottom of the respective lens assembly 600.

FIGS. 25A-25E show a further alternative lens assembly 630 with amovable stylus tip 632 for use in the MHMD goggles described herein. Asseen in FIG. 25B, the stylus tip 632 resides on the distal end of ashaft 634 positioned to slide axially within a stylus tube 636. Alinkage arm 638 pivotally attached at the proximal end of the shaft 634is also pivotally attached to a lever arm 640. The lever arm 640 ismounted to pivot within a frame 642, and has one of the finger pads 644on an end opposite the pivot point (not shown). A spring or other typeof return mechanism (not shown) is preferably included to maintain theequilibrium position of the lever arm 640, as seen in FIGS. 25A and 25B.When the wearer depresses the finger pad 644, as seen in FIGS. 25C and25E, the lever arm 640 raises up the end of the linkage arm 638 to whichit is connected, thus forcing the shaft 634 and stylus tip 632 distallyinto contact with the smartphone screen. Once again, a conductive pathfrom the stylus tip 632 to the finger pad 644 translates this movementinto a touch on the capacitive smartphone screen.

FIGS. 24 and 25 show the position of the styluses to the inside and topedge of each lens assembly, as opposed to at the bottom as in FIG. 22B.Again, the styluses are essentially hidden from the wearer's line ofsight—generally outside of their peripheral vision.

The significance of touching the smartphone screen can be to locate thelens assembly 600, thus setting the IPD distance. Alternatively, theability to touch the smartphone screen can be utilized as a, button,switch or prompt to make a decision with regard to software running inthe smartphone. For example, the first time a wearer puts on thegoggles, the smartphone may initiate an IPD calibration, wherein thewearer positions the lens assemblies 600 to his or her specification andinitiates the stylus touch. Subsequently, the smartphone software mayrequire inputs which can be translated through the stylus touch. Forexample, a number of YES or NO options can be presented to the wearer,wherein one touch means YES and two touches means NO (or a right sidetouch means YES and a left side touch means NO). Of course, there arenumerous other possibilities of such communication. Furthermore, asmentioned above, there may be more than one pair of touch stylusesprovided for the goggles which may allow for one dedicated pair (whichmay or may not be in constant contact with the screen of an insertedsmartphone) for IPD calibration and one or more other pairs forcommunicating decisions. Indeed, the use of two or more inputs greatlyenhances the user experience, much as a two button mouse is greatlysuperior to a single button mouse for interacting with a computer.

FIGS. 26A and 26B are front and rear perspective views, respectively, ofan exemplary remote controllers 512 for use with the MHMD goggles 500,while FIGS. 27A and 27B illustrate exemplary circuit boards 700 therein.As has been explained above, the exemplary MHMD goggles 500 desirablyinclude one or more remote controllers 512 detachably secured to anexternal surface of the main body 502. The remote controllers 512include internal motion sensors (not shown) and control buttons 702, aswell as a microprocessor (not shown) configured to communicativelycouple to the smartphone 572. It should be understood that “controlbuttons” refers to any type of devices manipulable by a user, such asbuttons, sliders, triggers, rotating rings or wheels, and joysticks,whether physical or virtual (i.e., touch screens). Furthermore, a cameralens 704 may be provided on a front end of the remote controllers 512.

As was described above, the remote controllers 512 may include one ormore 9-axis motion detection chip(s), although other numbers ofmotion-related axes may be used as desired. The remote controllers 512may communicate its current motion state (which may include orientation)to the smartphone 572 at a specified frequency, e.g., one or more timesper second, or when the motion state changes, e.g., by a specifiedamount.

The ability to attach and detach as well as positionally dock thecontrollers 572 to the main body 502 enables the user to easily keeptrack of the controller. While docked to the side of the main body 502,the controllers 512 can also be used in situations where the user wouldnot need to utilize the full features of the controller, as depicted inFIG. 18A, such as watching a 3D or spherical movie and using the controlbuttons 702 of the controller to play, pause or generally control theexperience. Preferably, each of the control buttons 702 is relativelylarge and has a distinct shape from the other control buttons so thatthe user can easily recognize and distinguish between them.

Furthermore, once the remote controllers 512 are docked onto the knownposition on the sides of the goggle main body 502, the system can thenuse the motion data from the controllers to track the user's head whileit is in motion. When docked, software on the smartphone 572 knows theorientation of the remote controller 512 based upon the dockingconfiguration (e.g. the remote controller 512 only docks in one positionon the goggles). The data generated by the remote controller 512 may beprovided in place of or in addition to data derived directly by asmartphone.

In addition, the docking mechanism can mechanically activate theheadtracking mode on the controller. For example, the bumps 574 on thesides of the goggle, under or near the docking clips 520 may depress thedocking switches 576 (see FIG. 21B). Of course, the bumps 574 represent“docking features” that may be formed by the main body 502 or by insertstherein, numerous possible configurations of which are contemplated. Forinstance, the bumps may be relatively rigid plastic inserts that are notcompressible like the soft body 502. When this occurs, softwareoperating on the remote controller 512 and/or smartphone 572 mayautomatically recognize that the remote control 512 has been docked withthe goggles. In one embodiment the docking mechanism presses the switch576 on the controller when the controller is in the dock allowing thecontroller to recognize that it is docked and take appropriate actionssuch as communicating its docked state to the system.

Although the docking switches 576 are shown relatively large andprotruding, they may also be smaller and recessed.

Similarly, other methods of detection may be employed in place of thedocking switches 576. Infrared, camera-based, light-based, magnetic,capacitive, proximity sensors and other systems used by the smartphoneand/or remote controller 512 may be used in order to detect that theremote controller 512 has been docked with the goggles. For example, acapacitive sensor may be exposed in a recess within the main body 502such that, when the remote controller 512 is docked, a small capacitivestylus touches the capacitive sensor thereby indicating that the remotecontroller 512 is docked. Similarly, infrared, camera-based,light-based, or proximity sensors may be employed to note when theremote views a particular light pattern, repeating light, light color,or similar indicator emitted by the smartphone and/or main body 502(e.g. through a particular recess in the side of the main body 502 thatcorresponds to a counterpart sensor in a remote controller 512) in orderto determine that the remote controller 512 is docked. Attachment to amagnet may close an exposed circuit on the main body 502 that indicatesthat the remote controller 512 is attached to the main body 502. Also,the controller 512 may include a mail USB jack that inserts into afemale port provide in the side of the body 502, which signals that thecontroller is docked and also provide a convenient means for data orpower transfer. These and various other docking detection methods may beemployed.

Once docked, and once recognized by either or both of the remotecontroller 512 and the smartphone 572, the remote controllers mayprovide orientation, location, motion, and rotation data to thesmartphone. The sensors or the integrated motion detection chip withinthe remote controllers 512 may be purpose-built for generatingmotion-related data. As a result of the increased use ofmotion-controllers (such as in the Wii and, now Wii U) and smartphonesuse of gyroscopes to determine screen orientation, direction and thelike, there are now very powerful integrated chips that are capable ofquickly providing and calculating device orientation, movement, androtation. However, in order to save costs the most powerful integratedchips are seldom integrated into smartphones. Instead, only thosesensors that provide some benefit, and only to the level that theyprovide that benefit, are typically incorporated into smartphones.

Because that very detailed data pertaining to orientation, location,movement, and rotation is desirable in a high-quality motion-detectingremote control, like remote controller 512, the integrated chips chosenfor integration into the remote controller 512 can be of the best, mostcost-effective quality. These chips can include (or have access to andalgorithms related to) one or more gyroscopes, gravitometers, compasses,magnetometers, cameras (both infrared and video) and other, similar,sensors used for determining orientation, location, movement androtation. Collectively, these are called “motion sensors” within thisapplication. Further, because the remote control in the presentapplication may be used in conjunction with a standard smartphone whichis not designed to perform such detailed calculations in order toprovide head-tracking data, the remote control provides an opportunityto offload some of that functionality at substantially reduced cost. Thedata generated by one or more of these remote controllers 512 may beextremely accurate, quickly generated, and transmitted to a smartphonefor action thereon. The remote controller 512 is shown as a remotecontrol device, but may instead be a fixed or detachable deviceincluding motion sensors and a processer that is only used inconjunction with the headset to augment the motion sensing capability ofa smartphone. Herein, these types of devices are also called remotecontrollers.

The process 200 shown in FIG. 13c exemplifies a typical interaction. Theprocess, generally, begins after one of the motion sensors is activatedbased upon a change in orientation of a remote controller attached tothe goggles. First, the sensors send the updated motion information inthe form of a signal to the mobile device (such as a smartphone) at 202.Because raw motion information can be complex, sensor fusion—the processof combining motion information from multiple sources (or sampled over aparticular time-frame)—may be performed on the data to derive motioninstructions that may be used to instruction video drivers orapplication software. Next, the mobile device sends the motioninstructions to an application, such as a virtual reality applicationdisplaying a virtual environment, at 203. Next, the application readsthe updated motion instructions at 204 and that information is used tochange the experience (such as updating the environment to reflect theupdated motion information) at 205.

In some cases, the remote may also be used to perform sensor fusion inaddition to providing raw sensor data or updated motion information to asmartphone 572. In such cases, the remote's integrated chips may obtainall location, motion, and rotation data and perform so-called “sensorfusion” to integrate that data into a current location, motion, androtation. That data may be handed off directly to the smartphone for usein rendering the current (or future) frames of video. Based upon thatraw data, the remote controller 512 may also perform predictivefunctions on the location, motion, and rotation data to thereby suggestfuture location, motion, and rotation of the goggle.

The remote controller 512 may perform motion sensor fusion in place ofor in addition to motion sensors in the smartphone 572, wherein thecontroller takes over some of the work for the smartphone. By relievingthe smartphone 572 of most tasks related to obtaining orientation,motion and rotation data, the smartphone apply its processing power toprocessor-intensive video rendering applications based upon the dataprovided by the remote.

Desirably, the remote controllers 512 may both equipped with a camera704 to provide additional video stream to the device used in conjunctionwith computer vision algorithms. The additional cameras 704 can be usedin conjunction with the camera on the smartphone 572 to provide a stereoimage of the environment. Providing even one controller 512 on a side ofthe main body 502 supplies an additional video stream, therebyfurthering enhancing the capabilities of the computer vision algorithmsby enabling the cameras of the smartphone 572 and remote control 12 towork in conjunction to provide a stereo image of the externalenvironment. Even more cameras, one on two, mounted remote controls 512and the smartphone 572 camera, may provide still more accuracy. Thecameras 704 on the controllers 512 may be RGB camera, depth cameras orsimply BW or UV cameras.

The detachable controllers 512 are also used to establish relativelocation of the system's motion components. Specifically, knowledge ofthe location and orientation of the controllers 512 allows the system tocalibrate the locations of the various motion components relative toeach other. Furthermore the system can then use the default positionsand orientations to provide positional and rotational offsets relativeto the default, thus allowing the system to track the motion of thecomponents relative to one another. This may, for example, act as a“reset” when motion tracking algorithms go awry. For example, the usermay apply and remove a controller 512 from his or her head to reset themotion tracking algorithm from a known starting point. This is usefulwhen the user removes the remote controller 512 from the headset withtheir hand, the system can then track the controller motion and applythat to a virtual rig of a human skeletal structure and compute theuser's virtual hand position based on the real world hand position.

Another configuration for the main body of the image in the goggles ofthe present application is in a collapsible form. For example, thevarious walls of the main body 502 illustrated above with respect toFIGS. 17-21 may be hingedly connected so that the body may be unfoldedand laid flat.

FIGS. 28A and 28B schematically illustrate a fully inflatable main body800 of a pair of MHMD goggles 802 of the present application. Separatelens assemblies 804 are fastened within a cavity 806 defined within theinflated body 800, as seen in FIG. 28B. The lens assemblies 804 are theonly rigid part, and as seen in FIG. 28A, the main body 800 whendeflated can be collapsed around the lens assemblies. An inflation valve808 is provided to convert the main body 800 from its deflated to itsinflated configuration. A smartphone retention pocket 810 is defined bythe inflated main body 800, much like what is described above. In thisembodiment, the lens assemblies 804 may be laterally movable, or theymay be fixed in place in a simplified version of the goggles.

FIGS. 29A and 29B show a partially inflatable embodiment of MHMD goggles900. A forward portion of the goggles 900 comprises a soft, compressiblematerial, such as the closed-cell phone described above. For example,the walls defining a smartphone retention slot 902 as well as channels(not numbered) for receiving separate lens assemblies 904 may be made ofthe soft, compressible material, or a more rigid material also asdescribed above. A rear portion of the goggles 900, such as sidewalls906 and a face-contacting lip 908 may be inflatable, and incorporate avalve 910. This configuration, the goggles 900 can be deflated andcompressed into a smaller brick shape for easy transport. With either afully or partially inflatable HMD, benefits include portability, lowerweight, price and ease of distribution at events.

FIG. 30A illustrates an alternative MHMD body 950 having a capacitivetouch slider 952 incorporated into one side wall 954. The slider 952 maybe mounted to slide vertically within a slit 956 formed in the body 950,or by a separate more rigid insert therein. FIG. 30B is a verticalsectional view showing the position of the slider 952 relative to asmartphone 958 retained within a retention pocket formed by the goggles.The slider includes a conductive stylus 960 that is positioned tocontact the display screen of the smartphone 958 so that a user maycontact the slider 952 and create a conductive path to the capacitivedisplay screen. Such a display screen slider 952 may be used tocommunicate instructions to the smartphone 958, such as controllingvolume, contrast, or other such features. Of course, more than one suchslider 952 may be provided, and the slider can be used to supplement theinput capacity of the two styluses mentioned above.

Closing Comments

Throughout this description, the embodiments and examples shown shouldbe considered as exemplars, rather than limitations on the apparatus andprocedures disclosed or claimed. Although many of the examples presentedherein involve specific combinations of method acts or system elements,it should be understood that those acts and those elements may becombined in other ways to accomplish the same objectives. Acts, elementsand features discussed only in connection with one embodiment are notintended to be excluded from a similar role in other embodiments.

As used herein, “plurality” means two or more. As used herein, a “set”of items may include one or more of such items. As used herein, whetherin the written description or the claims, the terms “comprising”,“including”, “carrying”, “having”, “containing”, “involving”, and thelike are to be understood to be open-ended, i.e., to mean including butnot limited to. Only the transitional phrases “consisting of” and“consisting essentially of”, respectively, are closed or semi-closedtransitional phrases with respect to claims. Use of ordinal terms suchas “first”, “second”, “third”, etc., in the claims to modify a claimelement does not by itself connote any priority, precedence, or order ofone claim element over another or the temporal order in which acts of amethod are performed, but are used merely as labels to distinguish oneclaim element having a certain name from another element having a samename (but for use of the ordinal term) to distinguish the claimelements. As used herein, “and/or” means that the listed items arealternatives, but the alternatives also include any combination of thelisted items.

It is claimed:
 1. A head mounted display system for use with a mobilecomputing device, comprising: a main body configured to be worn on ahuman head; a lens assembly within the main body comprising a first lensconfigured to focus vision of a wearer on a first area of a display ofthe mobile computing device when the mobile computing device is securedto the main body; and a first contact point disposed in a fixed positionrelative to the first lens and configured to contact a surface of thedisplay when the mobile computing device is secured to the main body, alocation at which the first contact point contacts the surface of thedisplay detectable by the mobile computing device and usable by themobile computing device to derive a position of the display relative tothe first lens.
 2. The head mounted display system of claim 1 whereinthe first contact point is brought into contact with the surface of thedisplay in response to securing the mobile computing device to the mainbody.
 3. The head mounted display system of claim 2 wherein the mobilecomputing device is secured to the main body by insertion of the mobilecomputing device into a retention pocket.
 4. The head mounted displaysystem of claim 1, wherein the first contact point comprises a stylushaving a length sufficient for a stylus tip to contact the surface ofthe display.
 5. The head mounted display system of claim 4, wherein thestylus tip is movable and is retracted from the display until a userpresses a button which advances the stylus tip into contact with thesurface of the display.
 6. The head mounted display system of claim 1,wherein the lens assembly further comprises: a second lens configured tofocus vision of the wearer on a second area of the display when themobile computing device is secured to the main body.
 7. The head mounteddisplay system of claim 6, wherein the second lens is disposed in afixed position relative to the first lens and the first contact point.8. The head mounted display system of claim 6, wherein the second lensis movable relative to the first lens and the first contact point, andthe head mounted display system further comprises a second contact pointdisposed in a fixed position relative to the second lens and configuredto contact the surface of the display when the mobile computing deviceis secured to the main body, a location at which the second contactpoint contacts the surface of the display detectable by the mobilecomputing device.
 9. The head mounted display system of claim 1, furthercomprising: a button accessible to a user's finger external to the mainbody; and a conductive path from the button to the first contact point.10. A head mounted display system, comprising: a main body configured tobe worn on a human head; a mobile computing device including a displaysecured to the main body; a lens assembly within the main bodycomprising a first lens configured to focus vision of a wearer on afirst area of the display; and a first contact point disposed in a fixedposition relative to the first lens and contacting a surface of thedisplay, wherein the mobile computing device detects a location wherethe first contact point contacts the surface of the display to determinea position of the display relative to the first lens.
 11. The headmounted display system of claim 10 wherein the mobile computing devicecontrols a position of data displayed on the display based upon theposition of the display relative to the at least one lens.
 12. The headmounted display system of claim 10 wherein the mobile computing devicederives horizontal and vertical positions of the first lens relative tothe display based upon the location at which the first contact pointcontacts the surface of the display.
 13. The head mounted display systemof claim 10 wherein the first contact point is brought into contact withthe surface of the display in response to securing the mobile computingdevice to the main body.
 14. The head mounted display system of claim 13wherein the mobile computing device is secured to the main body byinsertion of the mobile computing device into a retention pocket. 15.The head mounted display system of claim 10, wherein the first contactpoint comprises a stylus having a length sufficient for a stylus tip tocontact the surface of the display.
 16. The head mounted display systemof claim 15, wherein the stylus tip is movable and is retracted from thedisplay until a user presses a button which advances the stylus tip intocontact with the surface of the display.
 17. The head mounted displaysystem of claim 10, wherein the lens assembly further comprises: asecond lens configured to focus vision of the wearer on a second area ofthe display.
 18. The head mounted display system of claim 17, whereinthe second lens is disposed in a fixed position relative to the firstlens and the first contact point.
 19. The head mounted display system ofclaim 17, wherein the second lens is movable relative to the first lensand the first contact point, and the head mounted display system furthercomprises a second contact point disposed in a fixed position relativeto the second lens and configured to contact the surface of the display,a location at which the second contact point contacts the surface of thedisplay detectable by the mobile computing device.
 20. The head mounteddisplay system of claim 19 wherein the mobile computing device derivesan interpupillary distance based upon locations at which the firstcontact point and the second contact point contact the surface of thedisplay.
 21. The head mounted display system of claim 10, furthercomprising: a button accessible to a user's finger external to the mainbody; and a conductive path from the button to the first contact point.22. A head mounted display system for use with a mobile computingdevice, comprising: a main body configured to be worn on a human head; alens assembly within the main body comprising a set of lenses configuredto focus vision of a wearer on respective areas of a display of themobile computing device when the mobile computing device is secured tothe main body; and a contact point disposed in a fixed position relativeto each of the set of lenses and configured to contact a surface of thedisplay when the mobile computing device is secured to the main body, alocation at which the contact point contacts the surface of the displaydetectable by the mobile computing device and usable by the mobilecomputing device to derive a position of the display relative to the setof lenses; wherein the position is usable by a mobile computing deviceto adjust a position of an image displayed on the display.